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Educational evaluation has been generally ineffective 
in improving educational practice^ and many evaluations have 
dehuaanizing aspects* 1 conferenoe was held at Berkeley in April, 
1976, to consider this situation^ Participants included professional 
avaluators, teachers^ parents, students, and interest<sa others* One 
outcome was this hypothesisi The efficacy of information produced in 
an evaluation effort varies inversely with the number of 
organizational levels that the action the information describes is 
removed from the decision process the information; is intended to 
influence. In support^ it is noted that any information has a 
syntactic aspect^ a semantic aspect, and an effectiveness or 
influence aspects It is then argued that the syntactic aspect is 
violated in trying to detect small effects by aggregating data across 
organization levels I that greater error variance is produced in such 
an effort; that the most useful information semantically is that from 
the same organizational level as the user | and that inforaation from 
other levels is certain to be less motivating, Hecommendations 
include humanizing evaluation, involving the evaluated, removing 
racism and sexism from evaluation efforts use of more appropriatf 
evaluations by teachers^ increasing the relevance o£ evaluation tg 
decision makers^ and prganizing a public constltaeacy for change Qf 
evaluation pbilosophy and practice, <Author/MV) 
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1,. The lastae 

Cxitiai^ of edUfSational evalimtion has been inereaaing the past few 
years, Ona reasen is tiie general Inaf feotiveness of evaluation in improving 
eduoational practical and a seeond/ perhaps more dmniing one , is the de* 
hiama^lzing aspaots of many of the avaluationa whieh have been eondugted* 
anall groups of oonoerned avaluatora^ auoh as thm North Dakota Study Group, 
have attempted for several yaara to f oeus attantlon on oritioal issues such 
as the testing of blaok childran, and the inappropriate selection of children 
for special eduaationprograma. Generally, the aonoems of these groups have 
failed to dent the formal agendaa of orthodox forLuns, 

Khile most professional avaluators have debated their omi eaotario 
concerns in isolation from thosa affected by their work^ howev jr, a growing 
number of the parents, students , teachers and school atoinistrators have 
come to parcaive that educational evaluation, as it is currently practiced 
in many situations, may cause problems instead of helping to solve th^. 
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Thm situation calls for a ra-axamination. Stake Cundmt&d) cOTaaenta 

that? 

Hayba the whole idea of avaluation should go in the oircular file. 
School districts in Ohio, Illinois ^ and everywhere else have problems* 
Many of the problems are aggravated by people who oome to evaluate. 
We do not know whether or not evaluation is going to contribute more 
to the problCTi of education or more to the aolutions* 

This sort of oonGarn occurs at tiie same tto© that Uiare is incraAeing 

deaiand in the society for accountability, for demonstration by educators 

that they are in fact producing the outcomes they are supposed to*^hov?>sver 

vaguely dafinad thasa outcomes may be. Being more acoountabla meavsa 

producing hard data on toe "bottom line" to most people, and thus there 

appears to be a dilaiina between the need for more and batter infoOTation on 

the one hand ,and the aonsequencas of collecting it by aurrent methods on the 

other. 

All of this is a pracursor of c^re difficult dil^anas to come. Society 
is moving rapidly into toe so-called "Age of Inforffiation^" which is 
oharaaterized by expanding complexity in all aspects of life/ eloser inter*^ 
relatedness Miong groups and sooial syst«ns throughout the world, and 
enomous increases in the knowledge base. The dCTiand for more systCTiatio 
approaches to education and for greater use of technology and of inf oration 
on individuals in the educative process will grow, Movwieiit in this 
direction makes even more Imperative the concern of himanists /toat the basic 
values of society be presarved. Maintaining rights of privacy in the face 
of growing teehnalogy, for example # Is a fundamental and eKtraordinarlly 
difficult task. 

Riles (1973) notes that the problw in a sense is one of balancer 

^r society and our schools need a batter bal'mnce of values, That 
means the technicians, toe scientists^ toe engineers must concern 



themselves more with the htroan conse^ences of their rBsearah and 
achievements, And it means that the hmianistSr the artists, the 
educators must concern ttemselves more with the practical , realistic 
and technical planning neeessa:^ to achieve their purposes* 

It was against this baekground that thm conference reported in this 

paper was planned and conducted* ^'he purpose in general was to develop a 

clear ej^, understanding of the goals of evaluation in relation to the values 

of society and hopefully to create Uie impetue for integrating poaitiva 

change into thm evaluation proeass in aducation* 

II, Planning ^d Developing the Conference and Its Purposes 

The Conference's development can be traced to mid-1975, whan individuals 
and groups involved in evaluation in the San Francisco Bay Area began meeting 
with each other, and discovered that they shared ccamaon concerns. These 
meetings prOTioted an awareness of thm axistenca of a diversified motional 
constituency which was critical of the current state of aducational evaluation 
and which was either already articulating or was ready to articulate 
humanistic views within the field. 

The initiatora decided to attempt to convene a signif icant portion of 
this constituency in an atmosphere conducive to eKamination of the issues and 
to dialogue about the future o€ educational evaluation* Interest in current 
evaluation practices was intense enough to forge an ad hoc coalition of more 
than a do^en maters Mho worked wiyiout remuneration for mora than a year to 
develop a conference which would attract people from various perspectives, 
professional and life styles, and which would represent a meaningful 
alternativdi to the traditional conteKt in which evaluation has i^en discussed. 

Many Bay Mea ©valuators had been fiieling frustrated al^ut the de« 
hufflanizatlon of evaluation, as expressed in the evaluation of national Title I 

■■ ■ ■■■ VV'— ;:. 4.: ■ ■ - , " ■ ■ ■■- ■■■■ 
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and Follow Through Programs and of pujslic eduoation in general. At various 
conferences f when they tried to voice theaa f sruataf ationa ^ they were confronted 
with the absence of a professional ethic which could acc^TOTodate humanistic 
evaluation issues. 

The absence of a professional ethic did not aean absence of interest in 
the topic wittin the profession # however # At social hours and otter infoMal 
gatherings, numerous collea^es were found who were troi^led by such de*' 
hmnanizing evaluatiye trends as discounting all educational outcomes exQept 
those easily quantified tJurough standardised tasting. 

In the fall of 1975^ Paul Chaffee ^ of the California Council for the 
Humanities in Public Policy^ was actively meeting leading education everts 
in the Bay Area to discern the policy issues which were ci^rently being 
debated* The Council, a state-abased agency of i^e National Endowment for 
the Humanities I funds public prograimaing of policy Issues and asks grMtess 
to include academic hmianists in the dialogue* CSiaffee Indicated that tha 
Council would happily receive an application to fund a forrai in which the 
major issues relating to educational evaluation could be raised, 

Chaffee was in contact with Mick Hayder# an evaluation spacialist with 
Far West Laboratory in San Francisco* Together with Bill Baker of the 
alameda County School District , Rayder began eoniminicating with the Bay Area 
Consorti™, composed of school district representatives from six, Bay Area 
counties «id progrsuns such as Berkeley Es^erijaental Schools Project, Par 
Wast Laborato^ and University of California at Berkeley Field Services* 

In the ausmer of 197S, Consortiimi m^sabers envisioned a conference to 
follow the 1976 American ^uaational Eiaearch Association conference in 
San Francisco, Members felt ttiat AERA often biassed the real issues of 
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edueational evaluation. A grant appliaation for the (olloW'-up donferenaa was 
si^^ittad to the tolifornia Counall for thm H\mani ties and wae eueeaasful* 

The Berkeley Es^arimental Sehoola Project agreed to act as a co-^sponsor 
of the conference^ providing substantial financial and in-kind support, A 
variety of gro^aps, inoluding the school districts of San Francisco^ Richmond 
BSid iQameda County the University of California Field Services; The Institute 
for Scientific Analysis, Par West 3I*atorato^ff and the San Francisco Classroom 
Teachers Association provided in-kiftd services* At a later stage. Phi Delta 
Kappa supported the diss^iination of project material. 

To cons true tively attack the problens within a conference fr^ework, 
the planners decided on the following organizational principles s 

1* The conference should include concerned professionals frra a range 
of advocacy j^sitions in order to esti^lish a rigorous context in 
which to consider a^a eniotionally^chi^ged issues and to oonvey to 
professional evaluators that the mov^ent for evaluation ref om is 
widespread (if not yet clearly focused) , 
2 , Concerned individuals f ran the full variety of evaluation^ impacted 
life and career roles should be include ^ the ateosphere of the 
conference should be informal enough to permit each individual to 
feel qualified as an "eicpert" about his or her ovm evaluation- 
related ejqperience. ^e conference should not be d«iinated by 
evaluation professionals. 
3* The conferenoe-s forin^it should reflect an appreciation of pluralistic 
values, should be f IsKible and should by experienced-based in order 
to convey a sense of urgency around issues, 
4« The conference participants should fael^iat resources and e^ertise 
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suffieient to make a major inpaot on evaluation proeeduree were 
present at the conferenoa^ and that 'Uia question to be faced was# 
"What do wa do?'* rather than "What oan we make others do?" 
The Gonference wae held as scheduled in April 1976 • It represented the 

first B^jor atteinpt to engage evaluation analyzers in dialogue with each othe; 

with evaluation practitioners ^ with huMnist scholars^ and with funding 

sources* 



IIIp Conference Proceedings 

Several large group eKperiencea helped mmt the tone of the conferenca and 
provided a basis for broad scale interactions Mong participants* 

Itmaediately after registration on the opening day, the 150 conference 
participants gathered to view a frightening slide-tape prestotatlon concerning 
the year 2000. The narrator described a United States on the brink of total 
disintegration. The only hopej the viewers learned, lay in rigorous selection 
of the craatEi of the nation ^s youth for special school and advancement to leader 
ship positions* 

Each participant was given a packet consisting of information about 10 
hypothetical youthful candidates for special educational treatment in the year 
2000. Just two of the 10 candidates oauld be admitted to the special program, 
and participants ware asked to make a choiqa* The selection, it was stated, 
would determine flot just the educational future of individual candidates but 
also the general well baing of the nation and the allocation of considerable 
funding. Data on candidates Included demographic information, physical des-- 
criptions, intelligence quotients, sodial quotients, health evaluations, and 
information about goals. Interests, skills, and political beliefs. 

Participants worked in small groups, which made choices* Then the slide 
show recommenced and it was revealed that all of the applications had psuedo'- 
nymously described gifted individuals^ Most of the groups had excluded Albert 
Einstein CIQ of 82, very poor school adjustment , etct) , Isadora Duncan (low 
social quotient, behavior problem), Eleanor Roosevelt (erratic, withdrawn » 
seeks attention, falls often) and four others of the most illustrious indivi- 
duals of our times. 
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The point was well made that jud^ental evaluations — evaluation which 
separates good students and programs from bad --^ is dangerous. The consequent 
ces of misjudging or harmfully catergorlzlng human beings can be dlsasterous^ 
It la clear that the issue Is complex and that there are no quick answera. 

A second large--group experience was a preaentation by the Majumbe Dancers 
of black rhyttas from across the centuries. This group, through their per- 
formance^ illustrated that toerican educators and evaluators mist recognize 
that the dominant culture's definition of valid learning experiences and styles 
is not all--encompassing or^ indeed , sufficient. 

The third major large-group experience, a Saturday panel discussion on 
the topic, "Does Educational Evaluation Influence Public Policy? To What Ex* 
tent? At Whose Expense?" forced awareness of differing priorities for change. 
By the time the discussion was over ^ it was no longer possible to entertain the 
illusion that ©pposition to standardised testing is a sufficient coBmon plat^ 
form for reform of educational evaluation. Dispelling that illusion and recog^ 
nixing the diversity of vie^oince and interests that can be brought to bear 
upon evaluation by those outside of the evaluation --establishment" seem a 
necessary starting point for Improvement. 

Other experlerices involving conference participants as a group included 
Chinese ^ Mexican, and soul-^food meals, a reading of poetry about children, social 
hours and several suMaary sessions at which the progress of the conference was 
assessed. The conference schedule and format were adapted to meet the evolving 
conditions. As the participants faced issues from a variety of perspectives » 
personal interaction grew and abstraction decreased. This process was Inten-* 
sif led when participants entered small--group discussions, which were the other 
major activity of the conference. 



Twelve workgroups with abo^t i dozen participants each met four times 
curing the conference, with each session lasting about two hours, A 
conscious effort was made to provide for diversity of ^perience within each 
group. A facilitator from the conference planning comlttee provided initial 
impetus and structure as the sessions got undeway- 

The basic conference topetus toward refora of evaluation practice did not 
preclude a variety of separate agendas for such reform. For some, the work- 
groups were a frustrating experience, which dashed hopes of easy alliances 
toward "obvious" objectives* For others, the workgroups provided a vital ad- 
vance toward coping with the compl^ity of the forces involved In the evaluation 
controversy* 

In the conference's final session^ the individual workgroups reported 
to the whole* Virtually every group achieved concensus for its recoomendatlonBs 
and it was clear that the energy for reform had maintained Its force despite 
differing priorities. A widespread recognition had emerged that there are 
many problem areas, many routes to reform. 

Building on the workgroup process, a number of conference participants 
called an informal meeting one evening in an attempt to determine if a conference-^ 
wide concensus could be forged around any course of reform action. More than 
50 participants met late into the night i but at that meeting, as throughout the 
conference, those favoring a broadbased thrust toward short-term goals found 
themselves at odds with those emphasizing long term, systemic policy change* 

Those seeking to reform particular testing Instruments Md those opposed , 
to all standardised testing could not reconcile their differing priorities. 
Eventually, the participants agreed that the only feasible course was to . 
identify problem areas within educational evaluation and provMe a frMework 
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for cDnferance particlpantap and other interested persons, to pursue their 
various change agendas. 

The point should be made that the conference was not antl^evaluation — 
just anti educational evaluation which is useless or harmful or both* Indeed s 
there was recQgnition that evaluation which is positives i^^iich assists learners 
and educators to meet goals, Md which is humane in Its methods and outlook is 
essential. The facts that it is essential and that such a large praportion of 
educational evaluation to date has failed to meet these criteria make more 
crucial the critical considerations which emerged* If evaluation were not so 
important, then the issue would be staple | do away with lt| except perhaps as 
a scholastic exercise. This cannot and will not be the outcome, however , 8© 
the only meaningful course is to press to make evaluation in education more 
effective. 



IV. A Teehnlcal Outcomes 
Selecting Functional AnalymtB Unita 

A* The CroBS-^Levels Hypothesis 

In relation to increasing the effectiveness of educational evaluations 

a technical point emerged during the conference which seamed quite simple and 

logical at the times but which, on_further reflection of the authors ^ seeias to 

be more fundMiental that it first appeared* It might be stated in hypothesis 

form as follows s ^ 

The efficacy of Information produced in an evaluation ef for' t varies 
inversely with the number of organisational levels that the action the 
information describes is removed from the decision 
mation is intended to Influeace* ^ 

Evaluation is a process which produces information, as we all toow» and 
the information is intended to impact decisions, ^y evaluative information 
is produced by observations of the behavior of some entity p some unit of 
analysis. 

What the hypothesis says Is that Information resulting from evaluation 
will be more useful^ in the sense of Impacting decisions, the closer the d ex- 
cision maker is* prgaiilzatlonallyi to the analysis unit. Information describ- 
ing student performance in a classroom, according to this hypothesis, will be 
most useful to the classroom teacher^ less useful to the principal | still lass 
useful to the district superintendent, etc* . 

Looking at It from the other direction, the hypothesis says that, if you 
are a decision maker at some level in education and you need evaluative feedback 
to help you perform your job and meet your goals, the most useful feedback will 
be in terms of malysis units at your organisational level or at the levels v 
iBmediately adjacent "to you. If you are the US= Office of Education, for example 
and you are attemptihg to evaluate the results of some program — say .Title I — 



than the most useful information will coma from descriptions of organisational 
behavior at state governmant level, ' 

The hypothesis, considered In these terms, turns out to be very Interest- 
ing, for Ve are all aware of general failure to produce any definitive evalua- 
tions of Title I at the national level — ^ in spite of millions of dollars 
'3pent on the effort over some 12 years. In fact, a good many of the national 
evaluations of fader ally--funded programs, would have to be classified as fall^ 
ures - in terms of producing information helpful to deeision making. 

Why has there been failure, when some of the best brains around have given 
their bast to the effort? Is it because evaluation is a field still in its 
adolescense? Is it because the field is so fragmented and so much controversy 
exists within It? All of these relate to the problem, no doubt, but our hypo- ' 
thesis suggests a more basic reason. Most of the national evaluations have 
relied heavily on securing data on students at the classroom level, and then 
aggregating it up through various levels until it is finally put together in 
one large set nationally. If the hypothesis is valid, then one would predict 
in advance that such an evaluation effort would be of little value' 

B. Rationale 

The h3rpothesis emerged intuitively at the conference. After thinking 
through its consequences and realising th^i it could be an Important principle, 
the authors did some - background work to see if a theoretical rational could be 
built to support it p What follows in this section, therefore, was not produced 
at the conference i but is a post hoc attempt to demonstrate why the cross^levals 
hypothesis may be valid* We will be necessarily brief hare, with the intention 
of developing the argument more fully in the future* : : 



-13-. 

- 1. Information" and Its Purpos©a 

first need to define "Infomation" mora earefully* Aeeording to t^-^ 
Weaver (1963)* information can be conceived in three aspects — aocording : 
to its purely technical or syntactic qualities i according to its sema 
qualities^ or according to its effectiveness in influencing action. 

The technical or syntactic aspect deals with the engineering matters 
of transmission and receiving symbols , and the major concern is the accuracy . 
with which the transmission can be accomplished « In the semanti c aspect » 
the concern is with the understanding by the receiver of the meaning of the 
s3rmbols which have been transmitted^ The effectiveness or influence aspect " 
involves the capability of a message or set of data to motivate human action* 

These operate in a hierarchyi accurate transmission is essential to 
understanding, and understanding in turn is essential to effect iveness . 
Nocej however s that planning f or the use of . inf oriaatlon to support decision 
processes must start at the top* that la* with the determination of what is ^ 
needed for influence. 

The function of information is to reduce uncertainty related to specific 
needs* In organizations, information conveys objectives, plans, policies, 
and procedures to all components (subsystems). In this context, it serves 
both as a process activator to communicate requirements and as feedback 
to ensure that the communication has been received, understood^ and acted 
upon (Shrode and Volch, 1974, pp. 420^421). Information also performs 
another extremely crucial function of linking the organization to its environ- 
ment. Information^ therefore, reduces uncertainty about environmental challenges 
which- will :be encountered, about action alternatives for meeting these challenges 
about choices which have been made, about the extent to which choices have been : 
Imp 1 emen t ed and - ab ou t- ou t c ome s v— - — -—r. — r— — — ^_^„.„ 



Information can fall to perform its functions for several reasone. It 
can be tranamitted Inaccurately.. It can be misunderstood* It can be ig-^ 
nored. It can fall to reduce uncertainty^ or it can address issues which 
are not relevant* ' ■■ 

2, Applicable Theory 

Given this genera^, background * there are four theoretical propositions 
which support the cross-levels hypothesis of evaluation efficacy. 

... . 5 . 

Detecting Small Signals Amidst Great Noise , — ^In their review of the 

Handbook of Evaluation Research t the Stanford Evaluation Consortium (1976) 

comments on the difficulty of determining effects in social Interventions, 

The relationship between the processes of social intervention and 
evaluation has proven more complex and disconcerting than imagined. 
. • • . (The causes of social ills) are usually: multiple and only a 
few of them are affected by any one inte^entionV When many factors 
determine outcomes p the evaluator's task is to detect a small signal 
amidst a great deal of noise. Moreover^ when factors combine muiti* 
plicatively^ a standardi^intervention may be necessary but Insufficient, 
and there may be no signal to detect (pp, 12 and 13). 

Educational evaluators almost invariably work in complex situations and 
attempt to assess the effects of various types of social interventions. The 
factors determining outcomes are multiple, and there is noise in the system - 
with questionable measuring devices, for example, and \^th interference from 
other (unmeasured but relevant) variables. Under the best of conditions, the 
are unlikely to produce infomat ion which can definitively pinpoint cauge-and 
effect relationships. — 

There is a high probability, thereforej of failure in the first aspect of 
the Information hierarchy referred to previously, i.e. ^ in the syntactic 
sense. Incorrect 7 symbols are likely to be transmitted. As we shall see be- 
low, this situation is exacerbated when evaluative infomation must flow 



Greater Variance and Lost Meaning In C^oss'^Level Generalizationa ,"The term 
- "unit of analysis" iuggests Itself whe the hypothesis. In one 

sense, what we are saying is that the efficacy of educational evaluatlbni can 
be increased if evaluators will choose proper^ of analysis. This is%.n 

turn suggested by certain concepts in General System Theory. 
As Miller notes (1975, p- 351): [ 

The universe contains a hierarchy of systems » each higher level of system 
being composed of systems of lower levels. Atoms are composed of particles i 
molecules of atoms . . . . organisms of organs ^ groups (e.g. herds, flocks, 
families, teams j tribes) of organisms, organizations of groups, societies 
of organizations , and supranational systems of societies and organizations. 

Miller states that it Is Impbrtant to follow one procedural rule In 
to system level to avoid confusion. "Every discussion should begin with an Iden-- 
tificatlon of the level of reference, and the discourse should not change to 
another level without a specific stWement that this Is occurring (P. 21)" 

Another very important point is that "Cross-level generalisations or hypo^ 
theses which may apply to two or more levels will, ordinarily, have greater 
variance than the other sorts of generalizations, slnce^they Include variance 
among types and among individuals (p. 352)." 

Still another key consideration isi 

The more complex systems at higher levels manifest characteristics , more . 
than the sum of characteristics of the units, not observed at lower levels. 
. . . Significant aspects of living systems at higher levels will be neglected 
if they are described only in terms and dimensions used for their lower-' 
level subsystems and components (Miller* 1975, p. 353). 

If I start trying to consider the behavior of John Doe, teacher, I can des^ 
cribe the relevant variables at any number of levels. I could measure and talk 
about the biochemical reactions taking place in his cells. 1 would produce a 
terribly large quantity of data, of course, and I would experience considerable 
difficulty discerning from It the way Mr. Doe reacted to a student - s request for 
help. His biochemical processes are relevant! they are just too many levels re*- 
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moved from my level of re ferance^ however ^ for the semantic sense to be 
satisfied* I have great difficulty understanding what the data mean In tetms 
relevant to my concern, even if all of the data have been measured and trane^ 
mltted accurately. And all of it put together will not tell me all about 
the total behavior of the referent ayitemp Mr. Doe, 

Even if I had some great data-^reduclng mechanism which could integrate it 
all and perform this degree of cross-level aggregation* I would still have the 
problem of error variance being introduced at each level. In a situation where 
the relevant signals are already weak and difficult to detect, introduction of - 
more variance is likely to fatal to any chances of ever getting the Informa- 
tion to perform its Influencing function. 

Updating a^ System's View of Itself . As noted above , a maj or function of 
information is to reduce uncertainty about an organisation's. current state. 
There are certain '-key" variables or attributes^ the levels of which relate 
to success in achieveing goals, and persons in decision roles need to know 
periodically the status of these attributes so that they can take actions 
which will be goal--dlr acted, . 

The term "cybernetic" has been applied to this use of information, and the 
model is somewhat as follows: 

1, The system has recognized goals. - ^ 

2* Characteristics which relate to the achievement of these goals 
are understood, at least to some extent* 

3, These characteristics can be measured validly and transmitted 
accurately and meaningfully. 

4, Different decision makers are given infomation on characteristics 
relevant to their roles and levels of interest, . 

5m Baaed on the Information, they make decisions and take goal-oriented 



actions, 

&• New Information on the eharacteriatica is then fed baekj the 

results of the previous decision can be assessed^ and further 
.... ... ^. . 

actiong based on the new information can be taken* In this manner, 

the system is. guided to its goals. 

This model applies at all levels of self-organiglng systems^ and a key 
point is that the information must be relevant to the particular decision ma^ 
ker'e concernsp Suppose, for example, that I am a classroom teacher ^ and I 
am concerned about a child's learning progress* The model then becomes very 
similar to what has been called "individualized" or "customised" instruction. 

This is a type of Information which is likely to be highly useful to 
decision makeifs, and a major purpose of evaluation is to supply it* Seldom ' 
will information across more than one organization level be directly relevant 
to any decision procesSg Managers must deal at their own level of abstraction 
and Operation; those who concern themselves with details down the line will 
lose their effectiveness. 

The Stanford Evaluation Consortia (1976 ^ p. 18) suggests that producing 

this type of information should be a prime concern tn evaluation i 

Evaluation (in the new model) becomes a component of the evolving program 
itself J rather than disinterested monitoring, , , * Formal reports to 
outsiders are reduced in significance ^ and research findings become not 
conclusions but updatings of the system- s picture of itself* " 

Infonnatlon Which is Motivating * — To quote the Stanford group againj there 
is almost always a porcitical aspect in programs and situations with which edu- 
cational evaluation Is concerned* Evaluators like to proclaim themselves pro-- 
ducing data which will lead to rational decision making, and they are greatly ; 
disappo^inted when their outputs seem to be Ignored. They are often tempted to 
the conclusion that decisions about education are irrational or, at best, non- 



rational. This is a serloua miitake, for all decisions are rational from the 
point of view of the parson making them* it isj ratharj a matter of which 
priorities are in operation* "Certainly ^ the ©valuator who believes that the 
introduction of evaluation facts and findings will make an argument less pell* 
tical is dangerously deluded (Stanford , 1976, p. 12)." ' 

What Information does a decision maker use? Obviously s Information is: used 
which the decision maker perceives or has been conditioned to accept as Impbr^ 
tant to his own concerns in the matter at hand* This is something like deter- 
mining In behavioral analyais what will serve as a reinforcer. There is no 
doubt that human behavior is condiciosied; f^hat is a ralnforcer for mm may not 
be one for you, however, and specific reinforcers have different operational 
meaning for different persons* The SMie is true with different types of Infor 
matlon. 

The question at hand is what Information will act as a motivator of human 
action j that is, be sufficient in Weaver' s effectiveness or Influence sense * 
and it seems quite clear that the answer is closely related to the notlon^of- 
reinf or cement . That is, "the decision maker will be motivated by what he per^ 
ceives or has been conditioned to accept as in his ofm best interests, as re^ 
lated directly to the goals he Is seeking to achieve, as nirectly relevant to 
his own experiences* / 

This has to do inanediately with the cross^levels concept, for the simple 
fact is that, in most cases, himan beings will be less motivated by informa- 
tion removed from the level at which they are operating- To take an example 
with a time dimension, consider ^y concern for the different generations of 
my family. I take a great deal of Interest in myself. I am also quite In* 
terested in my children and my parents- Hy Interest lags a good deal when we 
shift to grandparents and grandchildren, and, the further we go. in generations 



from there, I have a difficult time getting very excited about it, I could 
care less^ in fact * what may happen to the tenth generation doTO the line in 
: 2177* ■■ ^' . . ■ ■■ . _ • 

C* Suggestions for Action ^ 

■To restate the hypothesis: 

The efficacy of information produced in an avaluation effort varies 
inversely with the number of organizational levels that the action the 
Information describes la removed from the decision process the Informa*- 
tlon is intended to influence- 
To the extent that this is valid ^ evalua tors would be well-advised to choose 
^analysis units at the pertinent decision level or at Immediately'-ad j acent levels , 
That is s they should produce information on the behavior or organizational units 
or systems at these levels. If you are in the US Office of Education, for 
eKample, concentrate on state--level actions and forget about trying to aggre-- 
gate up from classrooms* 

As corollaries to this major suggest ions we repeat two ideas from the 
conclusions of the Stanford Group (1976pp* 18)* 

•The evaluator should not concentrate on outcomes; ultimately, it may 
prove more profitable to study Just what was delivered and how people 
interacted daring the treatment process* 

* The evaluator should recognize (and act upon the recognition) that 
systems are rarely influenced by reports received in the mail* 
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V* Recommendations 

Pr&m workgroup reports and from consideration of the crosa-levela hypotHeslSj 
sets of conference recoamendations have been formulated, as follower 

A. Humanizing Evaluation 

1, To the fullest extent possible ^ evaluation should be descriptive, posl-- 
tive and ©ptlon-enhanclng rather than judgmental, negative and option decreasing 

2* Evaluation should be significant and applicable to the needs of those 
being evaluated* If data from an" evaluation program are not acted upon within 
a reasonable length of time, the evaluation should be terminated. 

3* Evaluators should be concerned with disrupting the educative process 
and should collect the minimum amount of data compatible with producing appro-- 
prlate and relevant information. 

4* Research and evaluation should be conceived as separate concerns, and ■ 
an attempt should be made to define the purposes and skills needed for eacha 
In addition to teclmlcal skills, for eKample^ skills in problem Identification 
and solving, interpersonal interaction and political awareness are necessary for 
carrying out a successful evaluation. , " ^ , 

- 5. Greater emphasis should be placed on gathering descriptive data about i 
school settings prior to the evaluation of the effect of particular changes or 
interventions, .. . .- ■ 

fia.lvalua tors should use a variety of analytical techniques (e,g« , anthro^ 
poldgical and convergent techniques) and coranlt themselves to the complexity of ^ 
their evaluative task, rather than relying on only .one level or type of analysis. 

6b.vT Evaluators and educators should take responsibility for updating evalua-- 
tidn^ procedures r v_ - — - ^ _ = _ 
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B, Involvltig the Evaluated 

1. The evaluated conmunity should have ultimate "ownership" of the evalua- 
tive proceaSp and should have the right to challenge evaluation results it feels 
are Invalid. 

2a. Evaluators should combine a professional approach with a receptivity , . 
to pluralism. For example* evaluators should utilize their technical training 
to insure the validity of their instruments ^ while at the s^e time inviting the 
input and participation of parents, students j and teachers in planning and carry 
Ing out the evaluation. The Early Childhood Education (ECE) evaluation program 
is commended because it provides for local input and individualization. 

2b. The role of the evaluator should be redefined so-^that he/she functions 
as an "ombudspersoni" i.e. , a person who represents the public interest^ is 
close to the public and is responsible to the public. 

3a. Evaluators should' strive for clarity to insure that ali affected under*- 
stand the meaning of evaluative discussions and reports. 

3b- Evaluators should make their findings widely available , especially to , 
people who might be able to learn from or apply these findings. 

C. Racism and Sexism in Evaluation 

la. Evaluators should actively address themselves to reaching and evaluating 
the ways in which racism and sexism influence chlldren-s education. 

lb. Evaluators should be aware of the diversity of ways in which indl^/idual 
children learn, and should develop evaluative and educative instruments which 
reflect this diversity. For example^ evaluators and educators should challenge ^ 
visual or linguistic materials which overrepresent males or confine f emales JtO- 
stereotypical roles, OO 



2a. .tfomen In evaluation^ralated fields should caucus to Insura that issues 
of racism and saxism are adequately addressed and that women's concerns are In^ 
tegrated Into discussions of edueation and social policy. 

2b. Women should be included in the dialogue at every level of evaluative 
decision-making. Being included means not only participating in the process of 
decision-making but also insuring that the terminology of decision-making is 
nonsexist* 

D. Teachers and Evaluation 

. . . . - . - , , , - ..... ... - „.■..■ 

1. Teachers should not allow tasting to determine education objectives or 
curriculum content. ^ i 

2« Teachers should utilize evaluative instrmaents and techniques which 
deemphasize comparison. For example, criterion referencing can be substituted 
for IQ testing; also j teachers can evaluateclaas progress in tems of their own 
teaching goals and a child's progress toward understanding materials presented^ 
rather than in terms of comparison between chlldren- 

E, Evaluation and Public Policy - : 

la.; The humanistic evaluation constltutency should analyze and understand 
the problems faced by those evaluatora who attempt to Mplement alternative ev- 
aluation techniques. Further ^ a concerted effort shotilil be made to legitimise 
alternative techniques In the eyes of the evaluation profession, the publlcand 
state and federal funding agencies. ; 

lb. Humanistic evaluators and educators should organize a public constit- 
uency for change of evaluation philosophy and practice. - z:..^- : 

2a, Evaluators should refuse to apply evaluative techniques which they be*- 
lieve to be ethically or professionally wrong, even if there is financial or 



/other preesura to apply these techniques. For example, the /^act that federal 
funds may appear to depend on the application of inadequate evaluative instruments 
should not be considered sufficient ground to apply sttch Instruments. 

2b, Evaluators and educators should avoid being stampeded by burM 
For example s when a directive is received from federal , state* county or district 
level* evaluators and educators should carefully determine whether the directive 
ISj In factj mandated— or ^whether other options exist. 

3. There should be concern with evaluating the quality of life in the schools. 
Including such indicators as the nimber of children who have visual-problems , 
who are hungry during the reading period and. so forth* Further, a statement should 
be developed that would discuss the social outcomes attributable to this quality 
of lack of quality of life in the schools. 

. F, Increasing the Relevance to Decision Processes - 

1, Evaluators should choose analysis units that are at the same organisational 
level or at immediately-adjacent levels to that of the decision process *the evalua^ 
tlon is intended to impact. 

2p Ivaluators should concentrate less on outcomes and more on what is de-- 
llvered in a program and on how people interact as the program is Implemented 
and operated. ^ . 

: : _ G* ^ Short Term Conference Follow-Up 

1, Conference participants should make a personal comltment to return to 
their work places and attempt to do whatever t hay can to make evaluation more 
ueeful and nwre himanistic* For ex^ple, they should stay in contact with each 
other to help solve coimaon problems- 
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2a* Conference participants should try to organiia local conferenees in their 
own areas that could Infom people about evaluation problems and alternatives 

and promote dialogue on the topic. 

_ 2c* A collection of documents or -a bibliography outlining what is already 
known about evaluation should be distributed to all thoee interested * as should 
a resource list of evaluative ropdels that represent alternatives to the use of 
standardized tests, ... 
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